{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting Housing Prices using Tensorflow + Cloud AI Platform\n",
    "\n",
    "This notebook will show you how to create a tensorflow model, train it on the cloud in a distributed fashion across multiple CPUs or GPUs and finally deploy the model for online prediction. We will demonstrate this by building a model to predict housing prices.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensorflow APIs\n",
    "<img src=\"assets/TFHierarchy.png\"  width=\"50%\">\n",
    "\n",
    "Tensorflow is a hierarchical framework. The further down the hierarchy you go, the more flexibility you have, but that more code you have to write. A best practice is to start at the highest level of abstraction. Then if you need additional flexibility for some reason drop down one layer. \n",
    "\n",
    "For this tutorial we will be operating at the highest level of Tensorflow abstraction, using the Estimator API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Steps\n",
    "\n",
    "1. Load raw data\n",
    "\n",
    "2. Write Tensorflow Code\n",
    "\n",
    " 1. Define Feature Columns\n",
    " \n",
    " 2. Define Estimator\n",
    "\n",
    " 3. Define Input Function\n",
    " \n",
    " 4. Define Serving Function\n",
    "\n",
    " 5. Define Train and Eval Function\n",
    "\n",
    "3. Package Code\n",
    "\n",
    "4. Train\n",
    "\n",
    "5. Deploy Model\n",
    "\n",
    "6. Get Predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1) Load Raw Data\n",
    "\n",
    "This is a publically available dataset on housing prices in Boston area suburbs circa 1978. It is hosted in a Google Cloud Storage bucket.\n",
    "\n",
    "For datasets too large to fit in memory you would read the data in batches. Tensorflow provides a queueing mechanism for this which is documented [here](https://www.tensorflow.org/guide).\n",
    "\n",
    "In our case the dataset is small enough to fit in memory so we will simply read it into a pandas dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#downlad data from GCS and store as pandas dataframe \n",
    "data_train = pd.read_csv(\n",
    "  filepath_or_buffer='https://storage.googleapis.com/spls/gsp418/housing_train.csv',\n",
    "  names=[\"CRIM\",\"ZN\",\"INDUS\",\"CHAS\",\"NOX\",\"RM\",\"AGE\",\"DIS\",\"RAD\",\"TAX\",\"PTRATIO\",\"MEDV\"])\n",
    "\n",
    "data_test = pd.read_csv(\n",
    "  filepath_or_buffer='https://storage.googleapis.com/spls/gsp418/housing_test.csv',\n",
    "  names=[\"CRIM\",\"ZN\",\"INDUS\",\"CHAS\",\"NOX\",\"RM\",\"AGE\",\"DIS\",\"RAD\",\"TAX\",\"PTRATIO\",\"MEDV\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Column Descriptions:\n",
    "\n",
    "1. CRIM: per capita crime rate by town \n",
    "2. ZN: proportion of residential land zoned for lots over 25,000 sq.ft. \n",
    "3. INDUS: proportion of non-retail business acres per town \n",
    "4. CHAS: Charles River dummy variable (= 1 if tract bounds river; 0 otherwise) \n",
    "5. NOX: nitric oxides concentration (parts per 10 million) \n",
    "6. RM: average number of rooms per dwelling \n",
    "7. AGE: proportion of owner-occupied units built prior to 1940 \n",
    "8. DIS: weighted distances to five Boston employment centers \n",
    "9. RAD: index of accessibility to radial highways \n",
    "10. TAX: full-value property-tax rate per $10,000 \n",
    "11. PTRATIO: pupil-teacher ratio by town \n",
    "12. MEDV: Median value of owner-occupied homes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Write Tensorflow Code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.A Define Feature Columns\n",
    "\n",
    "Feature columns are your Estimator's data \"interface.\" They tell the estimator in what format they should expect data and how to interpret it (is it one-hot? sparse? dense? continuous?).  https://www.tensorflow.org/api_docs/python/tf/feature_column\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "FEATURES = [\"CRIM\", \"ZN\", \"INDUS\", \"NOX\", \"RM\",\n",
    "            \"AGE\", \"DIS\", \"TAX\", \"PTRATIO\"]\n",
    "LABEL = \"MEDV\"\n",
    "\n",
    "feature_cols = [tf.feature_column.numeric_column(k)\n",
    "                  for k in FEATURES] #list of Feature Columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.B Define Estimator\n",
    "\n",
    "An Estimator is what actually implements your training, eval and prediction loops. Every estimator has the following methods:\n",
    "\n",
    "- fit() for training\n",
    "- eval() for evaluation\n",
    "- predict() for prediction\n",
    "- export_savedmodel() for writing model state to disk\n",
    "\n",
    "Tensorflow has several canned estimator that already implement these methods (DNNClassifier, LogisticClassifier etc..) or you can implement a custom estimator. Instructions on how to implement a custom estimator [here](https://www.tensorflow.org/guide/estimator) and see an example [here](https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/blogs/timeseries/rnn_cloudmle.ipynb).\n",
    "\n",
    "For simplicity we will use a canned estimator. To instantiate an estimator simply pass it what Feature Columns to expect and specify a directory for it to output to.\n",
    "\n",
    "Notice we wrap the estimator with a function. This is to allow us to specify the 'output_dir' at runtime, instead of having to hardcode it here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def generate_estimator(output_dir):\n",
    "  return tf.estimator.DNNRegressor(feature_columns=feature_cols,\n",
    "                                            hidden_units=[10, 10],\n",
    "                                            model_dir=output_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.C Define Input Function\n",
    "\n",
    "Now that you have an estimator and it knows what type of data to expect and how to interpret, you need to actually pass the data to it! This is the job of the input function. \n",
    "\n",
    "The input function returns a (features, label) tuple\n",
    "- features: A python dictionary. Each key is a feature column name and its value is the tensor containing the data for that Feature\n",
    "- label: A Tensor containing the label column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def generate_input_fn(data_set):\n",
    "    def input_fn():\n",
    "      features = {k: tf.constant(data_set[k].values) for k in FEATURES}\n",
    "      labels = tf.constant(data_set[LABEL].values)\n",
    "      return features, labels\n",
    "    return input_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.D Define Serving Input Function\n",
    "\n",
    "To predict with the model, we need to define a serving input function which will be used to read inputs from a user at prediction time. \n",
    "\n",
    "Why do we need a separate serving function? Don't we input the same features during training as in serving?\n",
    "\n",
    "Yes, but we may be *receiving* data in a different format during serving. The serving input function performs transformations necessary to get the data provided at prediction time into the format compatible with the Estimator API.\n",
    "\n",
    "returns a (features, inputs) tuple\n",
    "- features: A dict of features to be passed to the Estimator\n",
    "- inputs: A dictionary of inputs the predictions server should expect from the user"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def serving_input_fn():\n",
    "  #feature_placeholders are what the caller of the predict() method will have to provide\n",
    "  feature_placeholders = {\n",
    "      column.name: tf.placeholder(column.dtype, [None])\n",
    "      for column in feature_cols\n",
    "  }\n",
    "  \n",
    "  #features are what we actually pass to the estimator\n",
    "  features = {\n",
    "    # Inputs are rank 1 so that we can provide scalars to the server\n",
    "    # but Estimator expects rank 2, so we expand dimension\n",
    "    key: tf.expand_dims(tensor, -1)\n",
    "    for key, tensor in feature_placeholders.items()\n",
    "  }\n",
    "  return tf.estimator.export.ServingInputReceiver(\n",
    "    features, feature_placeholders\n",
    "  )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) Package Code\n",
    "\n",
    "You've now written all the tensoflow code you need!\n",
    "\n",
    "To make it compatible with Cloud AI Platform we'll combine the above tensorflow code into a single python file with two simple changes\n",
    "\n",
    "1. Add some boilerplate code to parse the command line arguments required for gcloud.\n",
    "2. Use the learn_runner.run() function to run the experiment\n",
    "\n",
    "We also add an empty \\__init__\\.py file to the folder. This is just the python convention for identifying modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "mkdir trainer\n",
    "touch trainer/__init__.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting trainer/task.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile trainer/task.py\n",
    "\n",
    "import argparse\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "from tensorflow.contrib.learn.python.learn import learn_runner\n",
    "from tensorflow.contrib.learn.python.learn.utils import saved_model_export_utils\n",
    "\n",
    "print(tf.__version__)\n",
    "tf.logging.set_verbosity(tf.logging.ERROR)\n",
    "\n",
    "data_train = pd.read_csv(\n",
    "  filepath_or_buffer='https://storage.googleapis.com/spls/gsp418/housing_train.csv',\n",
    "  names=[\"CRIM\",\"ZN\",\"INDUS\",\"CHAS\",\"NOX\",\"RM\",\"AGE\",\"DIS\",\"RAD\",\"TAX\",\"PTRATIO\",\"MEDV\"])\n",
    "\n",
    "data_test = pd.read_csv(\n",
    "  filepath_or_buffer='https://storage.googleapis.com/spls/gsp418/housing_test.csv',\n",
    "  names=[\"CRIM\",\"ZN\",\"INDUS\",\"CHAS\",\"NOX\",\"RM\",\"AGE\",\"DIS\",\"RAD\",\"TAX\",\"PTRATIO\",\"MEDV\"])\n",
    "\n",
    "FEATURES = [\"CRIM\", \"ZN\", \"INDUS\", \"NOX\", \"RM\",\n",
    "            \"AGE\", \"DIS\", \"TAX\", \"PTRATIO\"]\n",
    "LABEL = \"MEDV\"\n",
    "\n",
    "feature_cols = [tf.feature_column.numeric_column(k)\n",
    "                  for k in FEATURES] #list of Feature Columns\n",
    "\n",
    "def generate_estimator(output_dir):\n",
    "  return tf.estimator.DNNRegressor(feature_columns=feature_cols,\n",
    "                                            hidden_units=[10, 10],\n",
    "                                            model_dir=output_dir)\n",
    "\n",
    "def generate_input_fn(data_set):\n",
    "    def input_fn():\n",
    "      features = {k: tf.constant(data_set[k].values) for k in FEATURES}\n",
    "      labels = tf.constant(data_set[LABEL].values)\n",
    "      return features, labels\n",
    "    return input_fn\n",
    "\n",
    "def serving_input_fn():\n",
    "  #feature_placeholders are what the caller of the predict() method will have to provide\n",
    "  feature_placeholders = {\n",
    "      column.name: tf.placeholder(column.dtype, [None])\n",
    "      for column in feature_cols\n",
    "  }\n",
    "  \n",
    "  #features are what we actually pass to the estimator\n",
    "  features = {\n",
    "    # Inputs are rank 1 so that we can provide scalars to the server\n",
    "    # but Estimator expects rank 2, so we expand dimension\n",
    "    key: tf.expand_dims(tensor, -1)\n",
    "    for key, tensor in feature_placeholders.items()\n",
    "  }\n",
    "  return tf.estimator.export.ServingInputReceiver(\n",
    "    features, feature_placeholders\n",
    "  )\n",
    "\n",
    "train_spec = tf.estimator.TrainSpec(\n",
    "                input_fn=generate_input_fn(data_train),\n",
    "                max_steps=3000)\n",
    "\n",
    "exporter = tf.estimator.LatestExporter('Servo', serving_input_fn)\n",
    "\n",
    "eval_spec=tf.estimator.EvalSpec(\n",
    "            input_fn=generate_input_fn(data_test),\n",
    "            steps=1,\n",
    "            exporters=exporter)\n",
    "\n",
    "######START CLOUD ML ENGINE BOILERPLATE######\n",
    "if __name__ == '__main__':\n",
    "  parser = argparse.ArgumentParser()\n",
    "  # Input Arguments\n",
    "  parser.add_argument(\n",
    "      '--output_dir',\n",
    "      help='GCS location to write checkpoints and export models',\n",
    "      required=True\n",
    "  )\n",
    "  parser.add_argument(\n",
    "        '--job-dir',\n",
    "        help='this model ignores this field, but it is required by gcloud',\n",
    "        default='junk'\n",
    "    )\n",
    "  args = parser.parse_args()\n",
    "  arguments = args.__dict__\n",
    "  output_dir = arguments.pop('output_dir')\n",
    "######END CLOUD ML ENGINE BOILERPLATE######\n",
    "\n",
    "  #initiate training job\n",
    "  tf.estimator.train_and_evaluate(generate_estimator(output_dir), train_spec, eval_spec)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### 4) Train\n",
    "Now that our code is packaged we can invoke it using the gcloud command line tool to run the training. \n",
    "\n",
    "Note: Since our dataset is so small and our model is simply the overhead of provisioning the cluster is longer than the actual training time. Accordingly you'll notice the single VM cloud training takes longer than the local training, and the distributed cloud training takes longer than single VM cloud. For larger datasets and more complex models this will reverse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Set Environment Vars\n",
    "We'll create environment variables for our project name GCS Bucket and reference this in future commands.\n",
    "\n",
    "If you do not have a GCS bucket, you can create one using [these](https://cloud.google.com/storage/docs/creating-buckets) instructions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "GCS_BUCKET = 'gs://BUCKET_NAME' #CHANGE THIS TO YOUR BUCKET\n",
    "PROJECT = 'PROJECT_ID' #CHANGE THIS TO YOUR PROJECT ID\n",
    "REGION = 'us-central1' #OPTIONALLY CHANGE THIS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['GCS_BUCKET'] = GCS_BUCKET\n",
    "os.environ['PROJECT'] = PROJECT\n",
    "os.environ['REGION'] = REGION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run local\n",
    "It's a best practice to first run locally on a small dataset to check for errors. Note you can ignore the warnings in this case, as long as there are no errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.5.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python2.7/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "2018-03-05 18:56:25.561527: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "gcloud ai-platform local train \\\n",
    "   --module-name=trainer.task \\\n",
    "   --package-path=trainer \\\n",
    "   -- \\\n",
    "   --output_dir='./output'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run on cloud (1 cloud ML unit)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we specify which GCS bucket to write to and a job name.\n",
    "Job names submitted to the Cloud AI Platform must be project unique, so we append the system date/time. Update the cell below to point to a GCS bucket you own."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "jobId: housing_180305_185634\n",
      "state: QUEUED\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Job [housing_180305_185634] submitted successfully.\n",
      "Your job is still active. You may view the status of your job with the command\n",
      "\n",
      "  $ gcloud ai-platform jobs describe housing_180305_185634\n",
      "\n",
      "or continue streaming the logs with the command\n",
      "\n",
      "  $ gcloud ai-platform jobs stream-logs housing_180305_185634\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "JOBNAME=housing_$(date -u +%y%m%d_%H%M%S)\n",
    "\n",
    "gcloud ai-platform jobs submit training $JOBNAME \\\n",
    "   --region=$REGION \\\n",
    "   --module-name=trainer.task \\\n",
    "   --package-path=./trainer \\\n",
    "   --job-dir=$GCS_BUCKET/$JOBNAME/ \\\n",
    "   --runtime-version 1.15 \\\n",
    "   -- \\\n",
    "   --output_dir=$GCS_BUCKET/$JOBNAME/output\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run on cloud (10 cloud ML units)\n",
    "Because we are using the TF Estimators interface, distributed computing just works! The only change we need to make to run in a distributed fashion is to add the [--scale-tier](https://cloud.google.com/ml/pricing#ml_training_units_by_scale_tier) argument. Cloud AI Platform then takes care of distributing the training across devices for you!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "jobId: housing_180305_185638\n",
      "state: QUEUED\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Job [housing_180305_185638] submitted successfully.\n",
      "Your job is still active. You may view the status of your job with the command\n",
      "\n",
      "  $ gcloud ai-platform jobs describe housing_180305_185638\n",
      "\n",
      "or continue streaming the logs with the command\n",
      "\n",
      "  $ gcloud ai-platform jobs stream-logs housing_180305_185638\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "JOBNAME=housing_$(date -u +%y%m%d_%H%M%S)\n",
    "\n",
    "gcloud ai-platform jobs submit training $JOBNAME \\\n",
    "   --region=$REGION \\\n",
    "   --module-name=trainer.task \\\n",
    "   --package-path=./trainer \\\n",
    "   --job-dir=$GCS_BUCKET/$JOBNAME \\\n",
    "   --runtime-version 1.15 \\\n",
    "   --scale-tier=STANDARD_1 \\\n",
    "   -- \\\n",
    "   --output_dir=$GCS_BUCKET/$JOBNAME/output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run on cloud GPU (3 cloud ML units)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also works with GPUs!\n",
    "\n",
    "\"BASIC_GPU\" corresponds to one Tesla K80 at the time of this writing, hardware subject to change. 1 GPU is charged as 3 cloud ML units."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "jobId: housing_180305_183840\n",
      "state: QUEUED\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Job [housing_180305_183840] submitted successfully.\n",
      "Your job is still active. You may view the status of your job with the command\n",
      "\n",
      "  $ gcloud ai-platform jobs describe housing_180305_183840\n",
      "\n",
      "or continue streaming the logs with the command\n",
      "\n",
      "  $ gcloud ai-platform jobs stream-logs housing_180305_183840\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "JOBNAME=housing_$(date -u +%y%m%d_%H%M%S)\n",
    "\n",
    "gcloud ai-platform jobs submit training $JOBNAME \\\n",
    "   --region=$REGION \\\n",
    "   --module-name=trainer.task \\\n",
    "   --package-path=./trainer \\\n",
    "   --job-dir=$GCS_BUCKET/$JOBNAME \\\n",
    "   --runtime-version 1.15 \\\n",
    "   --scale-tier=BASIC_GPU \\\n",
    "   -- \\\n",
    "   --output_dir=$GCS_BUCKET/$JOBNAME/output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run on 8 cloud GPUs (24 cloud ML units)\n",
    "To train across multiple GPUs you use a [custom scale tier](https://cloud.google.com/ml/docs/concepts/training-overview#job_configuration_parameters).\n",
    "\n",
    "You specify the number and types of machines you want to run on in a config.yaml, then reference that config.yaml via the --config config.yaml command line argument.\n",
    "\n",
    "Here I am specifying a master node with machine type complex_model_m_gpu and one worker node of the same type. Each complex_model_m_gpu has 4 GPUs so this job will run on 2x4=8 GPUs total. \n",
    "\n",
    "WARNING: The default project quota is 10 cloud ML units, so unless you have requested a quota increase you will get a quota exceeded error. This command is just for illustrative purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting config.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile config.yaml\n",
    "trainingInput:\n",
    "  scaleTier: CUSTOM\n",
    "  masterType: complex_model_m_gpu\n",
    "  workerType: complex_model_m_gpu\n",
    "  workerCount: 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "jobId: housing_180305_183843\n",
      "state: QUEUED\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Job [housing_180305_183843] submitted successfully.\n",
      "Your job is still active. You may view the status of your job with the command\n",
      "\n",
      "  $ gcloud ai-platform jobs describe housing_180305_183843\n",
      "\n",
      "or continue streaming the logs with the command\n",
      "\n",
      "  $ gcloud ai-platform jobs stream-logs housing_180305_183843\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "JOBNAME=housing_$(date -u +%y%m%d_%H%M%S)\n",
    "\n",
    "gcloud ai-platform jobs submit training $JOBNAME \\\n",
    "   --region=$REGION \\\n",
    "   --module-name=trainer.task \\\n",
    "   --package-path=./trainer \\\n",
    "   --job-dir=$GCS_BUCKET/$JOBNAME \\\n",
    "   --runtime-version 1.15 \\\n",
    "   --config config.yaml \\\n",
    "   -- \\\n",
    "   --output_dir=$GCS_BUCKET/$JOBNAME/output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5) Deploy Model For Predictions\n",
    "\n",
    "Cloud AI Platform has a prediction service that will wrap our tensorflow model with a REST API and allow remote clients to get predictions.\n",
    "\n",
    "You can deploy the model from the Google Cloud Console GUI, or you can use the gcloud command line tool. We will use the latter method. Note this will take up to 5 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "gcloud config set ai_platform/region global"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Creating version (this might take a few minutes)......\n",
      "............................................................................................done.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "MODEL_NAME=\"housing_prices\"\n",
    "MODEL_VERSION=\"v1\"\n",
    "MODEL_LOCATION=output/export/Servo/$(ls output/export/Servo | tail -1) \n",
    "\n",
    "#gcloud ai-platform versions delete ${MODEL_VERSION} --model ${MODEL_NAME} #Uncomment to overwrite existing version\n",
    "#gcloud ai-platform models delete ${MODEL_NAME} #Uncomment to overwrite existing model\n",
    "gcloud ai-platform models create ${MODEL_NAME} --regions $REGION\n",
    "gcloud ai-platform versions create ${MODEL_VERSION} --model ${MODEL_NAME} --origin ${MODEL_LOCATION} --staging-bucket=$GCS_BUCKET --runtime-version=1.15"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6) Get Predictions\n",
    "\n",
    "There are two flavors of the AI Platform Prediction Service: Batch and online.\n",
    "\n",
    "Online prediction is more appropriate for latency sensitive requests as results are returned quickly and synchronously. \n",
    "\n",
    "Batch prediction is more appropriate for large prediction requests that you only need to run a few times a day.\n",
    "\n",
    "The prediction services expect prediction requests in standard JSON format so first we will create a JSON file with a couple of housing records.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing records.json\n"
     ]
    }
   ],
   "source": [
    "%%writefile records.json\n",
    "{\"CRIM\": 0.00632,\"ZN\": 18.0,\"INDUS\": 2.31,\"NOX\": 0.538, \"RM\": 6.575, \"AGE\": 65.2, \"DIS\": 4.0900, \"TAX\": 296.0, \"PTRATIO\": 15.3}\n",
    "{\"CRIM\": 0.00332,\"ZN\": 0.0,\"INDUS\": 2.31,\"NOX\": 0.437, \"RM\": 7.7, \"AGE\": 40.0, \"DIS\": 5.0900, \"TAX\": 250.0, \"PTRATIO\": 17.3}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will pass this file to the prediction service using the gcloud command line tool. Results are returned immediately!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PREDICTIONS\r\n",
      "[26098.3671875]\r\n",
      "[20871.384765625]\r\n",
      "\r\n",
      "\r\n",
      "Updates are available for some Cloud SDK components.  To install them,\r\n",
      "please run:\r\n",
      "  $ gcloud components update\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!gcloud ai-platform predict --model housing_prices --json-instances records.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "\n",
    "#### What we covered\n",
    "1. How to use Tensorflow's high level Estimator API\n",
    "2. How to deploy tensorflow code for distributed training in the cloud\n",
    "3. How to deploy the resulting model to the cloud for online prediction\n",
    "\n",
    "#### What we didn't cover\n",
    "1. How to leverage larger than memory datasets using Tensorflow's queueing system\n",
    "2. How to create synthetic features from our raw data to aid learning (Feature Engineering)\n",
    "3. How to improve model performance by finding the ideal hyperparameters using Cloud AI Platform's [HyperTune](https://cloud.google.com/ml-engine/docs/how-tos/using-hyperparameter-tuning) feature\n",
    "\n",
    "This lab is a great start, but adding in the above concepts is critical in getting your models to production ready quality. These concepts are covered in Google's 1-week on-demand Tensorflow + Cloud ML course: https://www.coursera.org/specializations/machine-learning-tensorflow-gcp"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
